Policy-Based Primal-Dual Methods for Convex Constrained Markov Decision Processes

We study convex Constrained Markov Decision Processes (CMDPs) in which the objective is concave and constraints are state-action occupancy measure. propose a policy-based primal-dual algorithm that updates primal variable via policy gradient ascent dual projected sub-gradient descent. Despite loss of additivity structure nonconvex nature, we establish global convergence proposed by leveraging hidden convexity problem, prove O(T^-1/3) rate terms both optimality gap constraint violation. When strongly measure, an improved O(T^-1/2). By introducing pessimistic term to constraint, further show zero violation can be achieved while preserving same for gap. This work first one literature establishes non-asymptotic guarantees methods solving infinite-horizon discounted CMDPs.